Probably the most complicated part of setting up a telecommunication system, including a BBS, is getting the cabling right. It seems so simple, yet it takes so many hours...
You should try out the cabling to the modem with a terminal program, not with HyperBBS. If you can talk to the modem and make calls using a terminal program (like Red Ryder or Microphone), HyperBBS will probably also work.
HyperBBS specifically avoids using the more esoteric 232 signals like DTR and DCD, just so you don't have to get that part of the cable right as well.